Multi-modal handwriting recognition correction

ABSTRACT

Systems, methods, and computer-readable media for processing electronic ink receive an electronic ink input; convert the electronic ink input to a first machine-generated object using handwriting recognition; display the first machine-generated object on a display; receive speech input; convert the speech input to a second machine-generated object using speech recognition; generate a list of machine-generated objects based on the electronic ink input, the list including the first machine-generated object and alternative machine-generated objects and functioning as a dictionary for converting the speech input; and replace the first machine-generated object with the second machine-generated object. The machine-generated objects may correspond to words, lines, and/or other groupings of machine-generated text. A user may confirm that the second machine-generated object should replace the first machine-generated object and the system will perform the replacement. The systems and methods may generate a list of alternative machine-generated object candidates to the first machine-generated object based on handwriting recognition of the electronic ink input alone or in combination with a statistical language model.

FIELD OF THE INVENTION

The present invention generally relates to systems, methods, andcomputer-readable media with computer-executable instructions storedthereon for processing electronic ink data in pen-based computingsystems and/or other computing systems.

BACKGROUND

Typical computer systems, especially computer systems using graphicaluser interfaces (GUIs), are optimized for accepting user input from oneor more discrete input devices, such as a keyboard for entering text,and a pointing device, such as a mouse with one or more buttons, foroperating the user interface. An example of such a GUI is the userinterface for the Windows® computer operating system (available fromMicrosoft Corporation of Redmond, Wash.). The ubiquitous keyboard andmouse interface provides for fast creation and modification ofdocuments, spreadsheets, database fields, drawings, photos, and thelike.

Recently, however, pen-based computing systems, such as tablet PCs andthe like, have been increasing in popularity. In pen-based computingsystems, user input advantageously may be introduced using an electronic“pen” or stylus (e.g., akin to writing with a pen or pencil on a pieceof paper). Indeed, in at least some pen-based computing systems, alluser input is capable of being entered and manipulated using anelectronic pen input device, and the user interface is fullycontrollable using only the electronic pen.

As pen-based computing systems are becoming more popular, users areincreasingly entering more data in the form of electronic ink. Inpen-based systems such as the tablet PC, users can write on the tabletPC just like they would write on a piece of paper. In many instances,however, users wish to convert the original electronic ink data tomachine-generated text, text suitable for use and manipulation byconventional word processing programs and other application programs.The tablet PC has handwriting recognition capabilities that converthandwriting to text. While the handwriting recognition in systems likethe tablet PC is very good, it is not foolproof.

Accordingly, there is a need in the art to provide an efficient,reliable correction mechanism for allowing a user to correct handwritingrecognition errors that result from the conversion of electronic ink totext. Such systems and methods would be particularly useful in pen-basedcomputing systems and other devices that accept input in the form ofelectronic ink and/or from an electronic pen or stylus and/or systems ordevices that display or otherwise process electronic ink data.

SUMMARY

Aspects of the present invention relate to systems and methods forprocessing electronic ink. Such systems and methods may include: (a)receiving electronic ink input (e.g., through an input device, such as adigitizer); (b) generating a list of machine-generated objects based onthe electronic ink input, the list including a first machine-generatedobject and alternative machine-generated objects; (c) converting theelectronic ink input to the first machine-generated object; (d)displaying the first machine-generated object; (e) receiving speechinput; (f) converting the speech input to a second machine-generatedobject (e.g., using a speech recognition engine), wherein the secondmachine-generated object is one of the alternative machine-generatedobjects and the list of machine-generated objects functions as adictionary used for converting the speech input; and (g) replacing thefirst machine-generated object with the second machine-generated object.The machine-generated objects may correspond to words, lines, and/orother groupings of machine-generated text.

In some aspects of the invention, a user may confirm that the secondmachine-generated object is correct; and the system will replace thefirst machine-generated object with the second machine-generated object.Systems and methods according to some aspects of the invention maydisplay the list of alternative machine-generated object candidates tothe first machine-generated object based on the electronic ink input. Inone aspect, the list of alternative machine-generated object candidatesincludes object candidates generated by a handwriting recognition enginebased on the electronic ink input and object candidates based on astatistical language model (e.g., unigram, bigram, trigram).

Additional aspects of the invention relate to computer-readable mediaincluding computer-executable instructions stored thereon for performingvarious methods and/or operating various systems, including systems andmethods like those described above.

These and other novel advantages, details, embodiments, features andaspects of the present invention will be apparent to those skilled inthe art from following the detailed description of the invention, theattached claims and accompanying drawings, listed herein, which areuseful in explaining the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features, and advantages of the presentinvention will be readily apparent and fully understood from thefollowing detailed description, taken in connection with the appendeddrawings, in which:

FIG. 1 illustrates a schematic diagram of a general-purpose digitalcomputing environment in which certain aspects of the present inventionmay be implemented;

FIG. 2 illustrates a pen-based personal computing (PC) environment inwhich certain aspects of the present invention may be implemented;

FIG. 3 shows an illustrative user interface utilized in a tablet PCinput panel for handwriting recognition correction.

FIG. 4 shows an illustrative user interface utilized in MicrosoftWindows Journal for handwriting recognition correction.

FIGS. 5 and 6 show an example flow diagram illustrating proceduresfollowed in at least some examples of this invention.

FIG. 7 depicts a screen with electronic ink data and the associatedmachine-generated text that resulted from handwriting recognition in anillustrative implementation of the present invention.

FIG. 8 depicts a screen including a correction user interface with atext candidate generated by speech recognition for correcting apreviously incorrectly recognized ink input according to an illustrativeembodiment of the present invention.

FIG. 9 depicts a screen including a correction UI after a user hasrequested a soft keyboard in connection with an illustrative embodimentof the present invention.

FIG. 10 depicts a screen including a correction user interface with adrop down menu of alternative text candidates according to anillustrative implementation of the present invention.

FIG. 11 depicts a screen showing how a parsing error may be correctedaccording to an illustrative embodiment of the present invention.

DETAILED DESCRIPTION

As described above, aspects of the present invention relate to systems,methods, and computer-readable media for processing electronic ink data.The following description is divided into sub-sections to assist thereader. The sub-sections include: Terms; General Description of VariousAspects of the Invention; Example Hardware; Example Systems, Methods,and Computer-Readable Media According to the Invention; and Conclusion.

I. Terms

The following terms are used in this specification and, unless otherwisespecified or clear from the context, the terms have the meaningsprovided below:

“Pen”—Any type of user input device useful in entering ink into and/orotherwise manipulating or controlling an electronic document. The terms“pen” and “stylus” are used interchangeably in this specification.

“Pen-Down Event”—An event that is initiated at the time a pen contacts adigitizer. Typically, a pen-down event will end at the time the penleaves the digitizer surface (also called a “pen-up event” in thisspecification).

“Computer-Readable Medium” means any available media that can beaccessed by a user on a computer system. By way of example, and notlimitation, “computer-readable media” may include computer storage mediaand communication media. “Computer storage media” includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information, such as computer-readableinstructions, data structures, program modules or other data. “Computerstorage media” includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology; CD-ROM, digital versatile disks (DVD)or other optical storage devices; magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices; or any othermedium that can be used to store the desired information and that can beaccessed by a computer. “Communication media” typically embodiescomputer-readable instructions, data structures, program modules orother data in a modulated data signal, such as a carrier wave or othertransport mechanism, and includes any information delivery media. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media, such as a wired network or direct-wiredconnection, and wireless media, such as acoustic, RF, infrared and otherwireless media. Combinations of any of the above should also be includedwithin the scope of “computer-readable media.”

II. General Description of Various Aspects of the Invention

Aspects of the present invention relate to systems and methods forprocessing electronic ink in pen-based computing systems and/or othercomputing systems that process, display, and/or otherwise handleelectronic ink data. One aspect of the invention relates to use speechto correct handwriting recognition errors. In another aspect, a spokenword can be recognized based on alternatives suggested by a handwritingrecognition engine and the recognized spoken word can replace thehandwritten recognized word on a display. In a further aspect, a userinterface is provided which accepts speech input and allows for in-placecorrection of machine-generated text based on handwriting recognition.

Aspects of the present invention relate to systems and methods forprocessing electronic ink. Such systems and methods may include: (a)receiving electronic ink input (e.g., through an input device, such as adigitizer); (b) generating a list of machine-generated objects based onthe electronic ink input, the list including a first machine-generatedobject and alternative machine-generated objects; (c) converting theelectronic ink input to the first machine-generated object; (d)displaying the first machine-generated object; (e) receiving speechinput; (f) converting the speech input to a second machine-generatedobject (e.g., using a speech recognition engine), wherein the secondmachine-generated object is one of the alternative machine-generatedobjects and the list of machine-generated objects functions as adictionary used for converting the speech input; and (g) replacing thefirst machine-generated object with the second machine-generated object.The machine-generated objects may correspond to words, lines, and/orother groupings of machine-generated text.

In some aspects of the invention, a user may confirm that the secondmachine-generated object is correct; and the system will replace thefirst machine-generated object with the second machine-generated object.Systems and methods according to some aspects of the invention maydisplay the list of alternative machine-generated object candidates tothe first machine-generated object based on the electronic ink input. Inone aspect, the list of alternative machine-generated object candidatesincludes object candidates generated by a handwriting recognition enginebased on the electronic ink input and object candidates based on astatistical language model (e.g., unigram, bigram, trigram).

Additional aspects of the invention relate to computer-readable mediaincluding computer-executable instructions stored thereon for performingvarious methods and/or operating various systems, including systems andmethods like those described above.

III. Example Hardware

FIG. 1 illustrates a schematic diagram of a general-purpose digitalcomputing environment that can be used to implement various aspects ofthe present invention. In FIG. 1, a computer 100 includes a processingunit 110, a system memory 120, and a system bus 130 that couples varioussystem components including the system memory 120 to the processing unit110. The system bus 130 may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. The system memory120 may include read only memory (ROM) 140 and random access memory(RAM) 150.

A basic input/output system 160 (BIOS), which contains the basicroutines that help to transfer information between elements within thecomputer 100, such as during start-up, is stored in the ROM 140. Thecomputer 100 also may include a hard disk drive 170 for reading from andwriting to a hard disk (not shown), a magnetic disk drive 180 forreading from or writing to a removable magnetic disk 190, and an opticaldisk drive 191 for reading from or writing to a removable optical disk192, such as a CD ROM or other optical media. The hard disk drive 170,magnetic disk drive 180, and optical disk drive 191 are connected to thesystem bus 130 by a hard disk drive interface 192, a magnetic disk driveinterface 193, and an optical disk drive interface 194, respectively.These drives and their associated computer-readable media providenonvolatile storage of computer-readable instructions, data structures,program modules, and other data for the personal computer 100. It willbe appreciated by those skilled in the art that other types ofcomputer-readable media that can store data that is accessible by acomputer, such as magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories (RAMs), read onlymemories (ROMs), and the like, may also be used in the example operatingenvironment.

A number of program modules can be stored on the hard disk drive 170,magnetic disk 190, optical disk 192, ROM 140, or RAM 150, including anoperating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices, such as akeyboard 101 and pointing device 102 (such as a mouse). Other inputdevices (not shown) may include a microphone, joystick, game pad,satellite dish, scanner, or the like. These and other input devicesoften are connected to the processing unit 110 through a serial portinterface 106 that is coupled to the system bus 130, but they also maybe connected by other interfaces, such as a parallel port, game port, ora universal serial bus (USB), and the like. Further still, these devicesmay be coupled directly to the system bus 130 via an appropriateinterface (not shown).

A monitor 107 or other type of display device also may be connected tothe system bus 130 via an interface, such as a video adapter 108. Inaddition to the monitor 107, personal computers typically include otherperipheral output devices (not shown), such as speakers and printers. Inone example, a pen digitizer 165 and accompanying pen or stylus 166 areprovided in order to digitally capture freehand input. Although aconnection between the pen digitizer 165 and the serial port interface106 is shown in FIG. 1, in practice, the pen digitizer 165 may bedirectly coupled to the processing unit 110, or it may be coupled to theprocessing unit 110 in any suitable manner, such as via a parallel portor another interface and the system bus 130 as is known in the art.Furthermore, although the digitizer 165 is shown apart from the monitor107 in FIG. 1, the usable input area of the digitizer 165 may beco-extensive with the display area of the monitor 107. Further still,the digitizer 165 may be integrated in the monitor 107, or it may existas a separate device overlaying or otherwise appended to the monitor107.

The computer 100 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer109. The remote computer 109 can be a server, a router, a network PC, apeer device or other common network node, and it typically includes manyor all of the elements described above relative to the computer 100,although for simplicity, only a memory storage device 111 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 112 and a wide area network (WAN)113. Such networking environments are commonplace in offices,enterprise-wide computer networks, intranets, and the Internet, usingboth wired and wireless connections.

When used in a LAN networking environment, the computer 100 is connectedto the local area network 112 through a network interface or adapter114. When used in a WAN networking environment, the personal computer100 typically includes a modem 115 or other means for establishing acommunications link over the wide area network 113, such as theInternet. The modem 115, which may be internal or external to thecomputer 100, may be connected to the system bus 130 via the serial portinterface 106. In a networked environment, program modules depictedrelative to the personal computer 100, or portions thereof, may bestored in the remote memory storage device.

It will be appreciated that the network connections shown are examplesand other techniques for establishing a communications link between thecomputers can be used. The existence of any of various well-knownprotocols such as TCP/IP, Ethernet, FTP, HTTP, UDP, and the like ispresumed, and the system can be operated in a user-server configurationto permit a user to retrieve web pages from a web-based server. Any ofvarious conventional web browsers can be used to display and manipulatedata on web pages.

Although the FIG. 1 environment shows one example environment, it willbe understood that other computing environments also may be used. Forexample, one or more examples of the present invention may use anenvironment having fewer than all of the various aspects shown in FIG. 1and described above, and these aspects may appear in variouscombinations and subcombinations that will be apparent to one ofordinary skill.

FIG. 2 illustrates a pen-based personal computer (PC) 201 that can beused in accordance with various aspects of the present invention. Any orall of the features, subsystems, and functions in the system of FIG. 1can be included in the computer of FIG. 2. The pen-based personalcomputer system 201 includes a large display surface 202, e.g., adigitizing flat panel display, such as a liquid crystal display (LCD)screen, on which a plurality of windows 203 is displayed. Using stylus204, a user can select, highlight, and write on the digitizing displayarea and thereby enter electronic ink data into the system. Examples ofsuitable digitizing display panels include electromagnetic pendigitizers, such as pen digitizers available from Mutoh Co. (now knownas FinePoint Innovations Co.) or Wacom Technology Co. Other types of pendigitizers, e.g., optical digitizers, may also be used. The pen-basedcomputing system 201 interprets gestures made using stylus 204 in orderto manipulate data, enter text as electronic ink, and executeconventional computer application tasks, such as creating, editing, andmodifying spreadsheets, word processing programs, and the like.

The stylus 204 may be equipped with buttons or other features to augmentits capabilities. In one example, a stylus 204 could be implemented as a“pencil” or “pen”, in which one end constitutes a writing portion andthe other end constitutes an “eraser” end, and which, when moved acrossthe display, indicates portions of electronic ink on the display thatare to be erased. Other types of input devices, such as a mouse,trackball, keyboard, or the like also could be used. Additionally, auser's own finger could be used as an input device, e.g., for selectingor indicating portions of the displayed image on a touch-sensitive orproximity-sensitive display. Consequently, the term “user input device,”as used herein, is intended to have a broad definition and encompassesmany variations on well-known input devices.

In various examples, the system provides an ink platform as a set of COM(component object model) services that an operating system and/or anapplication program can use to capture, manipulate, and store ink. Theink platform also may include a mark-up language including a languagelike the extensible markup language (XML). Further, the system may useDCOM (distributed component object model) as another implementation. Yetfurther implementations may be used including the Win32 programmingmodel and the Net programming model from Microsoft Corporation. Theseplatforms are commercially available and known in the art.

In addition to use with full performance pen-based computing systems or“tablet PCs” (e.g., convertible laptops or “slate” type tablet PCs),aspects of this invention can be used in conjunction with other types ofpen-based computing systems and/or other devices that accept data aselectronic ink and/or that accept process, or display electronic pen orstylus input, such as: hand-held or palm-top computing systems; personaldigital assistants; pocket personal computers; mobile and cellulartelephones, pagers, and other communication devices; watches;appliances; and any other devices or systems that include a monitor orother display device and/or a digitizer that presents printed orgraphical information to users and/or allows input using an electronicpen or stylus or which can process electronic ink collected by anotherdevice (e.g., a conventional desktop computer that can processelectronic ink collected by a tablet PC).

The invention now will be described in conjunction with the remainingfigures, which illustrate various examples of the invention andinformation to help explain the invention. The specific figures andinformation contained in this detailed description should not beconstrued as limiting the invention.

IV. Example Systems, Methods, and Computer-Readable Media According tothe Invention

As described above, aspects of the present invention relate generally tosystems and methods for processing electronic ink in pen-based computingsystems and other computing systems.

In some instances, handwriting recognition technology will not correctlyidentify the original ink words producing machine-generated text thatdoes not correctly correspond to the handwritten ink. Users may desireto correct these errors. Handwriting recognition of electronic ink inputand correction of machine-generated text is provided in, for example,the tablet PC input panel (TIP) found in the Tablet PC of MicrosoftCorporation, the “Convert Handwriting to Text” function in the actionsdrop down menu in Microsoft Windows Journal (Journal) and “Convert Inkto Text” function in a drop down menu in Microsoft OneNote (OneNote).TIP, Journal and Onenote each have different user interfaces for bothcalling the handwriting recognition engine and for correctinghandwriting recognition errors.

FIG. 3 shows an illustrative user interface 300 for TIP in a pen-basedcomputer system. TIP provides a writing pad 310 for the user to inputdata in electronic ink format. The electronic ink data 320 input to thewriting pad 310 is converted to machine-generated text by a handwritingrecognition engine in the processor of the computer system. In theexample of FIG. 3, a user has written the phrase, “Hello World” inelectronic ink on the writing pad 310 of the TIP user interface 300 of apen-based computing system. The system collects the ink data as the userinputs it, and then displays the sentence in the user's handwriting inelectronic ink 320 on the writing pad 310. The ink data can be collectedin any suitable manner and stored in any suitable format, such as inkserialized format (“ISF”). After the electronic ink data is collected,the system automatically sends the data to the handwriting recognitionengine, which will attempt to recognize the handwritten ink data andconvert the ink data to machine-generated text. Once the handwritingrecognition engine has processed the electronic ink input data, itreturns the recognized text to the application program (and/or otherwisemakes it available to the application program). In this example, themachine-generated text 330 is displayed in the window 340 of the openapplication, e.g., Microsoft Word. As shown, the electronic ink inputdata 320 has erroneously been recognized and converted to themachine-generated text 330 “Hello Wind”. To correct the handwritingrecognition error, a user can rewrite the word on the writing pad 310,i.e., input the electronic ink input data, and have the system recognizethe word again. This correction mechanism may not work because rewritingthe same phrase can result in another erroneous recognition.Alternatively, a user can correct the erroneous recognition using aphysical or soft keyboard to input the correct word.

FIG. 4 shows an illustrative user interface 400 for Journal in apen-based computer system. In FIG. 4, the user has input electronic inkdata on separate lines on Journal page 410 “Hello World” and “I amtesting handwriting recognition”. When the user desires to convert thehandwritten electronic ink input data to machine-generated text, theuser selects the ink to be converted and, from an “Actions” drop downmenu, chooses the function of “Convert Handwriting to Text” from themenu. In this example, the ink data “World” 415 is being considered forcorrection. Responsive to the user choosing the function of “ConvertHandwriting to Text” from the drop down menu in this example, the textcorrection dialog 420 is displayed on the user interface 400. As shown,the electronic ink input data is recognized and converted and displayedin the converted text region 430 of the text correction dialog 420. The“to be corrected” electronic ink input data is displayed in the inkregion 435 of the text correction dialog 420. Also, a list of possiblealternatives for the electronic ink input data displayed in the inkregion 435 is displayed in the alternative candidate list 440 of thetext correction dialog 420. To correct errors in the electronic inkinput data, the user can select one of the alternatives from thealternative list 440 or enter the word by using a keyboard in TIP.Should the, user wish to correct the text associated with otherelectronic ink input data, she can select the word she wants to correctby tapping on the word with the stylus in the converted text region 430.Responsive to the selection of another word, the electronic inkcorresponding to the selected word will appear in the ink region 435 andalternative candidates will be displayed in the alternative candidatelist 440.

In OneNote, the user selects the desired handwritten ink data to beconverted and chooses the “Convert Ink to Text” function from a Toolsdrop down menu. In response to selection of this function, Onenotereplaces the original ink with machine-generated text. To correct aword, a user can delete the erroneous word, rewrite the word and performanother ink to text conversion or the user can correct the erroneousword by using a keyboard, such as a soft keyboard.

Selection of the electronic ink data in the examples above could takeplace in any suitable or desired manner or as the result of any suitableor desired action. For example, touching a stylus on the display screen(a “pen-down event”) will result in selection of the specific word atthe location of the contact (e.g., if the stylus contacts the digitizerscreen within the “bounding box” occupied by a word, this action willresult in selection of that word). In other examples, a stylus may bedetectable by a digitizer while hovering adjacent to the digitizer'ssurface. For example, some styli are “active,” which in thisspecification means that they transmit, reflect, alter, or otherwisemodify energy, data, or other information transmitted by and/ordetectable by a digitizer when the stylus is located near the digitizersurface. As examples, a stylus may include circuitry or other structurethat receives energy emitted by the digitizer and/or alter magnetic orelectronic fields transmitted by and/or sensed by the digitizer, suchthat the presence of the stylus near the digitizer surface can be sensedby the digitizer due to changes in detected magnetic or electric fields.As another example, a stylus may transmit or reflect data, electrical ormagnetic fields, or other energy or information that is detectable bythe digitizer, at least when located near the digitizer surface. Anysuitable or desired way of sensing or detecting the presence of a styluswhen it is located near a digitizer surface may be used. Although astylus and digitizer combination may be designed to allow detection of astylus at virtually any reasonable position with respect to thedigitizer surface, in at least some conventional pen-based computingsystems with active pens, the stylus is initially detectable when it isabout one-half to two inches from the digitizer surface, and in someexamples, when it is approximately a half inch to an inch from thedigitizer surface. In such systems, if desired, selection of a word mayresult when the stylus hovers above the surface of the display screen,and the word immediately beneath or adjacent the tip of the stylusduring the hover action will be the word selected (e.g., if the hoveraction occurs over a bounding box of a word, that word will beselected).

As can be seen from the above examples, the user can either correct theword in two independent alternatives. First, by deleting the erroneousword and rewriting with the hope that the handwriting recognition enginewill properly recognize and convert the ink data. Obviously, thiscorrection mechanism may not work because rewriting the same phrase canresult in another erroneous recognition. Alternatively, the user can usea keyboard (e.g., a soft keyboard or physical keyboard) to correct aword. This requires the user to employ the keyboard, which may notalways be convenient, much less desired. Accordingly, it would behelpful to provide another methodology to correct handwritingrecognition/conversion errors.

FIG. 5 provides an illustrative flow diagram describing an exampleprocess that can be used to correct a handwriting recognition erroraccording to certain implementations of the invention. When the processstarts at step S500, the pen-based system receives an electronic inkinput. As described, ink data can be gathered in any suitable manner andin any suitable format without departing from the invention, includingin manners and formats that are conventional and known to those skilledin the art (such as ISF format). In step S502, an ink parser, preferablypart of the operating system 195, analyzes and classifies the electronicink input data as handwriting or drawing data, and if handwriting data,analyzes and groups the electronic ink input as text, e.g., word, lineand paragraph. For illustrative purposes, the present invention will bedescribed in terms of the electronic ink input data being handwriting,though the invention is not so limited.

Once the original electronic ink input data has been collected, analyzedand classified as handwriting (optionally, at various times while theink data is being entered), the ink parser can send the electronic inkdata to a handwriting recognition engine (e.g., in one implementationthe handwriting recognition in Microsoft WISP (Windows Ink ServicesPlatform) found in the Windows XP Tablet PC Edition operating system).The handwriting recognition engine converts the electronic ink input tomachine-generated text in step S504. In step S506, the system displaysthe machine-generated text corresponding to the electronic ink input ona display such as a Tablet PC display screen. In one implementation ofsteps S504 and S506 involving a Tablet PC, the user can select the“Tools” drop down menu, followed by selecting the option “Convert Ink toText” from the “Tools” menu. Responsive to the user commands, theelectronic ink input is converted to machine-generated text, which isdisplayed together (e.g., on top of or adjacent to) with the electronicink. Alternatively, the machine-generated text may be displayed in placeof the electronic ink. FIG. 7 shows an example of how the informationmay be displayed in steps S506. In this example, the originalhandwriting is softened and the converted text is shown on top of theoriginal handwriting.

After converting the electronic ink input to machine-generated text, theuser may select a portion of the machine-generated text for correctionin step S508. For example, to select the portion of text, the user maytap the area on the display with the stylus where the text is shown orhover over the portion of text with the stylus. The portion selected forcorrection may correspond to a word, letter, phrase, or sentence. Thefollowing discussion will describe an illustrative implementation of theinvention involving correction of a word. It will be appreciated by oneskilled in the art that the invention is not so limited, and theprinciples applied to a word can be applied to other portions of textfor correction.

When converting the electronic ink to text in step S504, the handwritingrecognition also generates a list of alternative machine-generated textcandidates, which might be correct candidates for the electronic inkinput. For example, the electronic ink input may be recognized as themachine-generated text “I”, but other probable alternativesmachine-generated text candidates that may be returned include “1”, “L”,“!”, “i”, “1” or “Z” due to their string similarity to “I”. In stepS509, the alternative-machine generated text candidates obtained fromhandwriting recognition for the selected word are defined as thedictionary for speech recognition. At this point speech input may beenabled.

A correction user interface can be provided to the user in step S510.The correction UI may automatically be provided or may be provided inresponse to a user input requesting the correction UI. The correction UImay at least in part take the form of a graphical user interface, whichmay include the ink representation of the selected portion forcorrection and also can, but does not have to, display the alternativemachine-generated text candidates. The alternative machine-generatedtext candidates can be displayed, for example as a drop down menu, atthe request of the user (e.g., by tapping on a portion of the UI).Initially, the list of alternative machine-generated text candidates mayinclude only alternative text candidates identified during thehandwriting recognition process as defined in step 509. In thisinstance, the alternative text candidates identified during handwritingrecognition can be used as the dictionary or vocabulary for speechrecognition. In an alternative implementation, the alternative textcandidates can include text candidates identified by a statisticallanguage model with or without the alternative text candidatesidentified by the handwriting recognition engine to be used as thedictionary for speech recognition. The alternative text candidates maybe displayed to the user in an order or with some other type ofrepresentation (e.g., symbol next to the candidate) indicating aconfidence measure that a respective alternative text candidatecorresponds to the electronic ink input.

Also, the user may request (e.g., by a pen down event on a portion ofthe correction user interface) that the initial dictionary for speechrecognition be expanded to include additional machine-generated textcandidates. The user may make such a request when the correct candidateis not found in the list of alternative machine-generated textcandidates identified during handwriting recognition. In oneimplementation, the vocabulary for speech recognition can be expanded toinclude additional alternative machine-generated text candidatesobtained from the statistical language model (LM). In oneimplementation, the text candidates obtained from the LM may bedisplayed together with list of text candidates identified duringhandwriting recognition with the confidence level of each candidatebeing represented to the user, for example by ranking most likelycandidate to least likely candidate from the top of the list to thebottom of the list.

An illustrative LM that may be used to generate the alternative textcandidates is the n-gram LM. In the LM, items of a unigram, bigram andtrigram may be contained. It will be appreciated to one skilled in theart that a myriad of LMs may be used to identify additionalmachine-generated text candidates and that the use of unigram, bigramand trigram is merely illustrative. Based on the machine-generated textcandidate and LM candidate list, a Viterbi search, known to thoseskilled in the art, may be performed to identify a path with the highestLM score among the candidate lists. Each node in the path can be assumedto be the most reasonable candidate (e.g., word in a grouping of wordssuch as a sentence).

Once the correction user interface (UI) has been presented to the userin step S510, several methodologies for correction of themachine-generated text candidate may be available including, amongothers, correction by speaking a text candidate, correction by inputthrough a soft keyboard, and correction by selecting a text candidatefrom a displayed list of alternative text candidates. Each of thesepossibilities will be discussed.

For correction by speaking a text candidate, the step S5 10 of providingthe correction user interface (UI) can include providing a mechanism foraccepting speech input. Speech input may be enabled by, for example, auser tapping the stylus on a microphone icon, such as shown in thecorrection UI shown in FIG. 8. A user can speak the correct word using aspeech input element such as a headset or microphone (e.g., built in tothe computer), which is associated with the pen-based computer. The userspeaks the word for correction into a speech input element and thespeech input is received at step S512.

In step S514, the speech input is converted into an alternativemachine-generated text candidate, the candidate being present in thealternative machine-generated text candidates list, which serves as thedictionary for speech recognition. In one implementation involving apen-based system from Microsoft Corporation, the speech recognitionengine processes the speech input using the Microsoft SAPI (speech API)5.1 with context-free grammar (CFG). The CFG grammar uses thevocabulary/dictionary for speech recognition created from the output ofthe handwriting recognition engine. When the user speaks the correctword, the speech recognition engine selects the alternativemachine-generated text candidate corresponding to the spoken word fromthe list of text candidates and replaces the incorrect word with themachine-generated text candidate recognized by the speech recognitionengine in step S516. The machine-generated text candidate recognized byspeech recognition can first be displayed, for example in the correctionUI of FIG. 8 prior to correction. Upon user confirmation, for example bytapping outside the correction UI in FIG. 8 (or acknowledging throughspeech or other input mechanism), the text candidate identified byspeech recognition can replace the incorrect word in the text sequenceon the display. In the example of FIG. 8, the incorrect text recognizedis the letter “Z”, and the user has spoken the text candidate “i”. Ifnecessary, the user could open the drop down menu by hovering or by wayof tapping the stylus on the down arrow in the correction UI. The dropdown menu might provide another alternative speech recognition textcandidate such as a capital “I” which the user could then select causingthe letter “Z” to be replaced with “I”.

At step S534 the correction process may be completed although it shouldbe understood that the process may be repeated or another methodologyfor correction may be used.

For correction by input through a soft keyboard, the step S510 ofproviding the correction UI provides a mechanism allowing a user toselect a soft keyboard for correction in step S518. Responsive to a userselection of a soft keyboard, a soft keyboard is on the display in stepS520. An illustrative example of a correction UI after the soft keyboardhas been displayed is shown in FIG. 9. In the example of FIG. 8, theincorrect text recognized is the letter “Z”. In steps S524, to correctthe text, the user can tap on a soft key(s) with the stylus to input thecorrect text and then tap on area outside the correction UI to cause thetext to be corrected by replacing the text with the soft key input. Atthis point the correction process has been completed at step S534.

For correction by selecting a text candidate from a displayed list ofalternative text candidates, the correction UI in step S510 can displaythe list of alternative text candidates or provide an interface allowingthe user to request display of the alternative text candidates in stepS526. Responsive to user selection causing a dropdown menu to open, thealternative text candidates provided by the handwriting recognitionengine can be displayed in step S528, such as in confidence order asshown in FIG. 10. From the list of alternative text candidates, in stepS530 the user can select the correct text candidate by, for example,tapping the stylus on the correct text candidate. Selection however, maytake other forms such as by speech recognition, for example by receivingspeech input and converting the recognized speech to the alternativetext candidate as described above in connection with correction byspeech recognition.

Also, the step of displaying the list of alternative text candidates instep S528 can include expanding or replacing the list of text candidatesby, for example, tapping on the down arrow button at the bottom of thecorrection UI shown in FIG. 10. In this instance, the LM may beactivated and an additional list of alternative text candidates can bedisplayed including the alternative text candidates suggested by the LM.In one implementation, the number of alternative text candidates in thelist may include twenty, though the number of candidates may be selectedto accommodate display real estate or to provide all the candidateswhich exceed a certain threshold confidence level. It should be notedthat obtaining LM text candidates can be processing intensive and thenumber of candidates may be limited based on the processing time neededto generate the LM candidate list.

In step S532, to correct the text, the user can tap on or hover over thetext candidate in the candidate list with the stylus to replace theincorrect text. It will be appreciated that selecting the text candidatein step S532 could involve the user speaking one of the alternative textcandidates on the alternative text candidate list with or without theuser confirming that the speech recognized input is correct to replacethe incorrect text. At step S534 the correction process may be completedalthough it should be understood that the process may be repeated oranother methodology for correction may be used.

Of course, FIGS. 5 and 6 merely constitute an example of steps that maybe taken in accordance with at least some examples of the invention.Those skilled in the art will recognize that the content of the specificsteps may be changed, the steps may be performed in a different order,additional steps may be added, and/or existing steps may be omitted orcombined without departing from the invention.

Parsing errors can occur when the handwriting recognition enginerecognizes electronic ink input data as two or more words where the inkinput data was intended to represent one word or when electronic inkinput data is recognized as one word where the input was intended torepresent more than one word. Implementations of the invention allow forcorrection of parsing errors. User can select the correct grouping ofelectronic input text data by, for example circling (e.g., defining aregion encompassing) the appropriate text with the stylus to eitherseparate or group the appropriate text. Thus, when two words have beenrecognized when the user intended the ink strokes of the recognizedwords to be a single word, the user may circle the ink strokes making upboth words with the stylus to indicate that they should be groupedtogether and considered as one word. Similarly, if two words have beenrecognized as one word the user can circle the ink strokes correspondingto a portion of the recognized word to indicate that the circled inkstrokes in the corresponding recognized word and the remaining inkstrokes in the corresponding recognized word should be recognized asseparate words.

FIG. 11 shows a user interface where the written phrase “am testing” hasbeen recognized as “antedating”. In the FIG. 11 example, the user hascircled the ink strokes with the stylus corresponding to the word “am”.The selected ink strokes can be highlighted in some manner, such asshown in FIG. 11, once the user lifts the stylus from the digitizer. Atthis juncture, the user can provide an input (e.g., tapping the styluson or hover over the highlighted portion, providing a voice command)causing the handwriting recognition engine to convert the selected inkstrokes to text and the other non-highlighted input strokes(corresponding to “testing”) to text. The machine-generated textcandidate is displayed. The operation for grouping strokes is analogousin that the grouped strokes are highlighted, the user provides an inputrequesting the handwriting recognition engine to convert the strokes totext, and the resulting machine-generated text candidate is displayed.

Below is a more detailed explanation of applying the LM to generate thepreviously described candidate lists. Given a word sequence,

the handwriting recognition engine gives each word w_(i) somecandidates. Assume the user wants to correct the word w_(i), a word listwhich the speech recognition engine can use must be generated. The ideais that the words in the dictionary can be filtered using the stringsimilarity, and then the LM can be used to calculate the probabilitythat the word appears in the context. Finally, the weighted sum of thescore in the LM and the score in string similarity are used for rankingthe words in the dictionary for the speech recognition engine. In oneimplementation, twenty candidates may be selected, though this numbercan vary. One factor that may influence the number of candidates isdisplay real estate.

The candidates for the word w_(i) given by the handwriting recognitionengine may not contain the correct word, but the sub segments of thecorrect word are usually the sub segments of these candidate words. Thusstring similarity information may be used to determine candidates. Astring matching process may be conducted using the Dynamic ProgrammingString Alignment algorithm. For example, assume the word “these” and theword “horses”, the result of string matching is as below.

The count of the matched characters is 3. The candidates of w_(i) givenby the handwriting recognition engine are represented as w_(i,j). To aword d_(k) in the dictionary, the similarity between d_(k) and w_(i,j)is defined as${s_{sim}\left( {d_{k},w_{i,j}} \right)} = {\log\quad\left( \frac{c_{match}\left( {w_{i,j},d_{k}} \right)}{{{len}\left( w_{i,j} \right)} + {{len}\left( d_{k} \right)}} \right)}$The c_(match)(w_(i,j),d_(k)) is the count of the matched charactersbetween d_(k) and w_(i,j) (considering order), the len(w_(i,j)) is thecharacter count of word w_(i,j), and the len(d_(k)) is the charactercount of word d_(k). To the word “these” and the word “horses”,${s_{sim}\left( {{{}_{}^{}{}_{}^{}},{{}_{}^{}{}_{}^{}}} \right)} = {{\log\quad\left( \frac{3}{5 + 6} \right)} \approx {- 0.564271}}$The score of d_(k) in string similarity iss _(sim)(d _(k))=max(s _(sim)(d _(k) ,w _(i,j)))

Theoretically speaking, every word d_(k) in the dictionary is acandidate of w_(i) for the speech recognition engine. But if the stringsimilarity between d_(k) and any w_(1,j) is very small, it is notnecessary to use d_(k) as the candidate of w_(i) for the speechrecognition engine, because it can be assumed that although thehandwriting recognition engine cannot give the correct word, it can givesome (enough) characters of the correct word. For example, if thecandidate of w_(i) that the handwriting recognition engine provides is“these”, the word “is” is almost impossible to be the correct word ofw_(i) because the similarity between the two words is very small.Applying this method, the computing complexity can be reduced.Furthermore, because the count of the common characters between d_(k)and any w_(i,j) (not considering order) is equal to or greater than thecount of the matched characters between d_(k) and w_(i,j), if there isno common character between d_(k) and w_(i, j) or the count of thecommon characters is very small, the string similarity between d_(k) andany w_(i,j) does not need to be calculated.

For each word d_(k) in the dictionary following the string matchingprocess, the LM model

can be applied to calculate the probability of occurrence of the wordd_(k) in the word sequence using the Language Model.P(w _(i−2) w _(i−1) d _(k) w _(i+1) w _(i+2))=max(p(w _(i−2,j) _(i−1) w_(i−1,j) _(i−1) d _(k) w _(i+1,j) _(i+1) w _(i+2,j) _(i+2) ))The score of the word d_(k) in the LM iss _(LM)(d _(k))=log(P(w ¹⁻² w _(i−1) d _(k) w _(i+1) w _(i+2)))

To generate the word list for the speech recognition engine, the scoreof the LM and the score of the string similarity can be combined andweighted to obtain the weighted score,s(d _(k))=s _(LM)(d _(k))+λ*s _(sim)(d _(k))λ is the weight of the score of string similarity. Using the weightedscore, the confidence level associated with each word can be determinedrelative to other words and the words can be ranked based on theprobability that they would correspond to the electronic ink input. Inone implementation, the top twenty words may be selected.

In sum the steps for generating candidates using the LM is 1) the wordsin the dictionary are filtered using the string similarity score; 2) theLM score is calculated; and 3) the words are ranked using the weightedsum of the LM score and the string similarity score, and the top (e.g.,twenty) ranked words are selected as machine-generated text candidates.

Those skilled in the art will recognize that aspects of this inventioncan be applied to any type of electronic ink data and machine-generatedobjects, including handwritten text in any language and written or readin any direction without departing from the invention. Additionally,aspects of this invention can be applied to recognition and processingof other types of data and machine-generated objects without departingfrom the invention, such as data and elements in charts, diagrams,graphs, flowcharts, etc.; musical data or symbols; mathematical orscientific data or symbols; drawings; etc.

Finally, the present invention also relates to computer-readable mediaincluding computer-executable instructions stored thereon for performingvarious methods and/or for use in various systems, including thosesystems and methods described above. The computer-readable media mayconstitute computer-executable instructions stored on the variousspecific examples of computer-readable media described above.

V. CONCLUSION

Various examples of the present invention have been described above, andit will be understood by those of ordinary skill that the presentinvention includes within its scope all combinations and subcombinationsof these examples. Additionally, those skilled in the art will recognizethat the above examples simply exemplify various aspects of theinvention. Various changes and modifications may be made withoutdeparting from the spirit and scope of the invention, as defined in theappended claims.

1. A method comprising: receiving electronic ink input; generating alist of machine-generated text candidates based on the electronic inkinput, the list including a first machine-generated text candidate andalternative machine-generated text candidates; converting the electronicink input to the first machine-generated text candidate; displaying thefirst machine-generated text candidate; receiving speech input;converting the speech input to second machine-generated text, whereinthe second machine-generated text is one of the alternativemachine-generated text candidates and the list of machine-generated textcandidates functions as a dictionary used for converting the speechinput; and replacing the first machine-generated text candidate with thesecond machine-generated text.
 2. The method of claim 1, wherein thefirst machine-generated text candidate is a word.
 3. The method of claim1, wherein the first machine-generated text candidate is a portion of aword.
 4. The method according to claim 1, further comprising receivinginput selecting the first machine-generated text candidate prior toreceiving the speech input.
 5. The method according to claim 4, whereinthe selecting includes touching a user input device to a digitizerscreen at a location corresponding to the first machine-generated textcandidate.
 6. The method according to claim 4, wherein the firstmachine-generated text candidate is a group of words or part of a word.7. The method according to claim 1, further including displaying thelist of machine-generated text candidates prior to receiving the speechinput.
 8. The method according to claim 7, wherein said step ofdisplaying the alternative machine-generated text candidates furtherincludes displaying the alternative machine-generated text candidates inthe list in an order based on a confidence level that each alternativemachine-generated text candidate corresponds to the electronic inkinput.
 9. The method according to claim 1, wherein the alternativemachine-generated text candidates include machine-generated textcandidates based on the electronic ink input generated by a handwritingrecognition engine.
 10. The method according to claim 9, wherein thealternative machine-generated text candidates include machine-generatedtext candidates based on the electronic ink input generated inaccordance with a statistical language model.
 11. The method accordingto claim 10, further comprising displaying the machine-generated textcandidates generated by the handwriting recognition engine andsubsequently displaying the machine-generated text candidates generatedin accordance with the statistical language model.
 12. The methodaccording to claim 11, further comprising receiving input requesting thedisplay of the machine-generated text candidates generated in accordancewith the statistical language model while displaying themachine-generated text candidates generated by the handwritingrecognition engine.
 13. The method according to claim 1, wherein thealternative machine-generated text candidates include text candidatesbased on the electronic ink input generated by a statistical languagemodel.
 14. The method according to claim 1, wherein the step ofconverting the speech input to the second machine-generated textincludes determining if the speech input corresponds to one of thealternative machine-generated text candidates; and converting the speechinput to the corresponding alternative machine-generated text candidatewhen the speech input corresponds to the alternative machine-generatedtext candidate.
 15. The method according to claim 1, wherein furthercomprising the step of receiving an input confirming that the secondmachine-generated text candidate should replace the firstmachine-generated text candidate prior to performing said step ofreplacing.
 16. A computer-readable medium including computer-executableinstructions stored thereon for performing the method of claim
 1. 17. Amethod, comprising: receiving electronic ink input; generating a list ofmachine-generated objects based on the electronic ink input, the listincluding a first machine-generated object and alternativemachine-generated objects; converting the electronic ink input to thefirst machine-generated object; displaying the first machine-generatedobject; receiving speech input; converting the speech input to a secondmachine-generated object, wherein the second machine-generated object isone of the alternative machine-generated objects and the list ofmachine-generated objects functions as a dictionary used for convertingthe speech input; and replacing the first machine-generated object withthe second machine-generated object.
 18. The method according to claim17, further comprising the step of receiving input confirming that thesecond machine-generated object should replace the firstmachine-generated object prior to performing said step of replacing. 19.The method according to claim 17, further comprising receiving inputselecting the first machine-generated object prior to receiving thespeech input.
 20. The method according to claim 17, further comprisingdisplaying a list of alternative machine-generated objects on thedisplay prior to receiving the speech input.
 21. The method according toclaim 20, wherein said step of displaying the alternativemachine-generated objects further includes displaying the alternativemachine-generated objects in the list in an order based on a confidencelevel that each alternative machine-generated object corresponds to theelectronic ink input.
 22. The method according to claim 17, wherein thealternative machine-generated objects include objects based on theelectronic ink input generated by a handwriting recognition engine. 23.The method according to claim 22, wherein the alternativemachine-generated objects include machine-generated objects based on theelectronic ink input generated in accordance with a statistical languagemodel.
 24. The method according to claim 23, further comprisingdisplaying the machine-generated objects generated by the handwritingrecognition engine and subsequently displaying the machine-generatedobjects generated in accordance with the statistical language model. 25.The method according to claim 24, receiving input requesting the displayof the machine-generated objects generated in accordance with thestatistical language model while displaying the machine-generatedobjects generated by the handwriting recognition engine.
 26. The methodaccording to claim 17, wherein the alternative machine-generated objectsinclude machine-generated objects based on the electronic ink inputgenerated by a statistical language model.
 27. The method according toclaim 17, wherein the step of converting the speech input to the secondmachine-generated object includes determining if the speech inputcorresponds to one of the alternative machine-generated objects; andconverting the speech input to the corresponding alternativemachine-generated object when the speech input corresponds to thealternative machine-generated object.
 28. A computer-readable mediumincluding computer-executable instructions stored thereon for performingthe method of claim
 17. 29. A system comprising: a display; a firstinput adapted to receive electronic ink input; a second input adapted toreceive speech input; and a processor programmed and adapted to: (a)convert the electronic ink input to first machine-generated text usinghandwriting recognition; (b) display the first machine-generated text onthe display; (c) convert the speech input to second machine-generatedtext using speech recognition; (d) generate a list of machine-generatedtext candidates based on the electronic ink input, the list including afirst machine-generated text candidate and alternative machine-generatedtext candidates and functioning as a dictionary for converting thespeech input; and (e) replace the first machine-generated text candidatewith the second machine-generated text.
 30. The system according toclaim 29, wherein the first machine-generated text is a word.
 31. Thesystem according to claim 29, wherein one of the first or second inputsis adapted to receive a confirmation input that the secondmachine-generated text is correct before replacing the firstmachine-generated text with the second machine-generated text.
 32. Thesystem according to claim 29, wherein the first input is further adaptedto receive a selection of a portion of the first machine-generated textfor correction.
 33. The system according to claim 32, wherein theselected portion is the entire first machine-generated text.
 34. Thesystem according to claim 29, wherein said processor is furtherprogrammed to display the alternative machine-generated text candidateson the display prior to the second input receiving the speech input. 35.The system according to claim 29, wherein said processor is furtherprogrammed to display the alternative machine-generated text candidateson the display in an order in the list based on a confidence level thatthe respective alternative machine-generated text candidate correspondsto the electronic ink input.
 36. The system according to claim 29,wherein said processor is programmed to generate the alternativemachine-generated text candidates based on the electronic ink inputusing a handwriting recognition engine.
 37. The system according toclaim 36, wherein said processor is programmed to generate thealternative machine-generated text candidates based on the electronicink input using a statistical language model.
 38. The system accordingto claim 37, wherein the first input is further adapted to receive arequest to display the machine-generated text candidates generated inaccordance with the statistical language model while displaying themachine-generated text candidates generated by the handwritingrecognition engine.
 39. The system according to claim 29, wherein saidprocessor is programmed to generate the alternative machine-generatedtext candidates based on the electronic ink input using a statisticallanguage model.
 40. The system according to claim 29, wherein saidprocessor is further programmed to determine if the speech inputcorresponds to one of the alternative machine-generated text candidates;and convert the speech input to the corresponding alternativemachine-generated text candidate when the speech input corresponds tothe alternative machine-generated text candidate.
 41. A systemcomprising: a display; a first input adapted to receive an electronicink input; a second input adapted to receive speech input; and aprocessor programmed and adapted to: (a) convert the electronic inkinput to a first machine-generated object using handwriting recognition;(b) display the first machine-generated object on the display; (c)convert the speech input to a second machine-generated object usingspeech recognition; (d) generate a list of machine-generated objectsbased on the electronic ink input, the list including the firstmachine-generated object and alternative machine-generated objects andfunctioning as a dictionary for converting the speech input; and (e)replace the first machine-generated object with the secondmachine-generated object.
 42. The system according to claim 41, whereinone of the first or second inputs is adapted to receive a confirmationinput that the second machine-generated object is correct beforereplacing the first machine-generated object with the secondmachine-generated object.
 43. The system according to claim 41, whereinsaid processor is further programmed to display the alternativemachine-generated objects on the display prior to the second inputreceiving the speech input.
 44. The system according to claim 41,wherein said processor is programmed to generate the alternativemachine-generated objects based on the electronic ink input using ahandwriting recognition engine.
 45. The system according to claim 44,wherein said processor is programmed to generate the alternativemachine-generated objects based on the electronic ink input using astatistical language model.
 46. The system according to claim 45,wherein the first input is further adapted to receive a request todisplay the machine-generated objects generated in accordance with thestatistical language model while displaying the machine-generatedobjects generated by the handwriting recognition engine.
 47. The systemaccording to claim 41, wherein said processor is programmed to generatethe alternative machine-generated objects based on the electronic inkinput using a statistical language model.
 48. The system according toclaim 41, wherein said processor is further programmed to determine ifthe speech input corresponds to one of the alternative machine-generatedobjects; and convert the speech input to the corresponding alternativemachine-generated object when the speech input corresponds to thealternative machine-generated object.